fix(server): reconcile crash-frozen running turns to interrupted on boot (#5) - #13
Merged
Merged
Conversation
… boot Fixes #5. When the backend exits ungracefully (SIGKILL/crash/OOM, no SIGTERM), the Effect finalizer a clean quit runs (ProviderService.runStopAll -> adapter.stopAll() -> a session-set that settles running turns to "interrupted") never fires. In-flight turns are left frozen at projection_turns.state="running" with active_turn_id still set; on a later resume the provider reports the cut-off turn as cancelled/aborted, and auto-resume never recovers them. At fresh boot no turn is actually live (adapter session maps start empty; runtime rows carry no pid/epoch), so every leftover in-flight turn is a crash remnant. Add a one-shot boot reconciler that mirrors the graceful path: for each such thread it dispatches a stopped `thread.session.set` command (through the event log, not a direct projection write), settling the running turn to the resumable "interrupted" state and clearing active_turn_id. Wired as a startup phase before reactors start and before commands are accepted. Scope: reconciling to a resumable interrupted state satisfies the issue's primary acceptance. Auto-resume RE-ARMING (a crash marker + boot producer to auto-continue the turn) is left as a documented follow-up. Adds CrashRecoveryReconciler.test.ts (precondition, reconcile, idempotency, multi-thread selectivity, event-log backing) and extends serverRuntimeStartup and autoResume/guards tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #5. When the desktop backend exits ungracefully (SIGKILL / crash / OOM — no SIGTERM), in-flight agent turns are left frozen at
projection_turns.state = "running"withprojection_thread_sessions.active_turn_idstill set. Nothing reconciles them on next launch, so on a later resume the provider reports the cut-off turn ascancelled/aborted, and auto-resume never recovers them. A clean quit does the right thing via an Effect finalizer (ProviderService.runStopAll→adapter.stopAll()→ a session-set that settles running turns tointerrupted); only ungraceful exit was broken.Approach
At fresh boot no turn is actually live — adapter session maps start empty and runtime rows carry no pid/epoch — so every leftover in-flight turn in the read model is by definition a crash remnant. The fix adds a one-shot boot reconciler that mirrors the graceful path exactly:
apps/server/src/orchestration/Layers/CrashRecoveryReconciler.ts—reconcileInterruptedTurnsOnBootreads the read model (ProjectionSnapshotQuery.getSnapshot()), finds threads with a leftover in-flight turn (latestTurn.state === "running", or sessionstatus ∈ {running, starting}, or non-nullactiveTurnId), and for each dispatches athread.session.setcommand withstatus: "stopped",activeTurnId: null(mirroring whatProviderRuntimeIngestiondispatches onsession.exited). This settles the running turn to the canonical resumableinterruptedstate and clearsactive_turn_id.serverRuntimeStartup.tsas a startup phase beforereactors.startand before command-ready is signalled, so no reactor or user command observes the stale state, and the (now-unblocked)ProviderSessionReapercan idle-sweep the runtime row.Key correctness property: reconciliation goes through the event log (a dispatched command), never a direct projection write — projections are a pure function of the event log and a direct write would be reverted on rebuild. It is fully error-swallowed (
nevererror channel) so a bad thread or persistence hiccup can't block boot.Scope (deliberate)
Reconciling to a resumable
interruptedstate satisfies the issue's primary acceptance ("or at minimum settled to a resumableinterruptedstate"). Auto-resume re-arming (auto-continuing the turn) needs a new crash marker + a boot-time producer to distinguish crash-interrupted from user/stop-interrupted turns; that's a documented, separate follow-up (noted in code comments) rather than bundled here.Tests (TDD — red first)
CrashRecoveryReconciler.test.ts(new, integration over the real engine + projection pipeline + in-memory sqlite): precondition (turnrunning, sessionrunning,active_turn_idset) → reconcile → turninterruptedwithcompleted_at, sessionstopped,active_turn_idnull,reconciledCount === 1; idempotency (second run is a no-op); multi-thread selectivity (only crashed threads change; already-settled untouched); event-log backing (a newthread.session-setevent is appended, surviving a rebuild).serverRuntimeStartup.test.ts(extended): the reconcile phase runs beforereactors.start/ command-ready, and a dispatch failure is swallowed (startup still completes).t3x/autoResume/guards.test.ts(extended): a settledinterrupted+stoppedthread is notthreadIsProgressing(documents resumability).Verification
src/orchestration/+src/t3x/autoResume/→ 255 pass.tsgo --noEmit→ exit 0, no errors.vp fmt --checkon changed files → clean.🤖 Generated with Claude Code